Method, system and computer readable medium for an intelligent search workstation for computer assisted interpretation of medical images

ABSTRACT

A method, system and computer readable medium for an intelligent search display into which an automated computerized image analysis has been incorporated. Upon viewing an unknown mammographic case, the display shows both the computer classification output as well as images of lesions with known diagnoses (e.g., malignant vs. benign) and similar computer-extracted features. The similarity index used in the search can be chosen by the radiologist to be based on a single feature, multiple features, or on the computer estimate of the likelihood of malignancy. Specifically the system includes the calculation of features of images in a known database, calculation of features of an unknown case, calculation of a similarity index, display of the known cases along the probability distribution curves at which the unknown case exists. Techniques include novel developments and implementations of computer-extracted features for similarity calculation and novel methods for the display of the unknown case amongst known cases with and without the computer-determined diagnoses.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority under 35 U.S.C. § 119(e) to U.S.provisional patent applications Ser. No. 60/180,162, filed on Feb. 4,2000 and Ser. No. 60/207,401, filed on May 30, 2000 and also priorityunder 35 U.S.C. § 120 to U.S. patent application Ser. No. 09/773,636,filed on Feb. 2, 2001, now U.S. Pat. No. 6,901,156, each of which isincorporated by reference herein.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates generally to the field of computer-aided diagnosisand image display workstations. It describes a method, system andcomputer readable medium that employs an intelligent search workstationfor the computer assisted interpretation of medical images. Upon viewingan unknown medical case, the workstation shows both computer analysisoutput as well as images of cases with known diagnoses (e.g., malignantvs. benign) and similar computer-extracted features. The similarityindex used in the search can be chosen by the radiologist to be based ona single feature, multiple features, or on the computer estimate of thelikelihood of disease (such as malignancy in breast cancer). The presentinvention also generally relates to computerized techniques forautomated analysis of digital images, for example, as disclosed in oneor more of U.S. Pat. Nos. 4,839,807; 4,841,555; 4,851,984; 4,875,165;4,907,156; 4,918,534; 5,072,384; 5,133,020; 5,150,292; 5,224,177;5,289,374; 5,319,549; 5,343,390; 5,359,513; 5,452,367; 5,463,548;5,491,627; 5,537,485; 5,598,481; 5,622,171; 5,638,458; 5,657,362;5,666,434; 5,673,332; 5,668,888; 5,732,697; 5,740,268; 5,790,690;5,832,103; 5,873,824; 5,881,124; 5,931,780; 5,974,165; 5,982,915;5,984,870; 5,987,345; and 6,011,862; as well as U.S. patent applicationSer. Nos. 08/173,935; 08/398,307 (PCT Publication WO 96/27846);08/536,149; 08/562,087; 08/900,188; 08/900,189; 08/900,191; 08/900,361;08/979,623; 08/979,639; 08/982,282; 09/027,468; 09/027,685; 09/028,518;09/053,798; 09/092,004; 09/121,719; 09/131,162; 09/141,535; 09/156,413;09/298,852; and 09/471,088; PCT patent applications PCT/US99/24007; andPCT/US99/25998; and U.S. provisional patent application 60/160,790;filed on Jan. 18, 2000, all of which are incorporated herein byreference.

The present invention includes use of various technologies referencedand described in the above-noted U.S. Patents and Applications, as wellas described in the references identified in the appended LIST OFREFERENCES by the author(s) and year of publication and cross-referencedthroughout the specification by numerals in brackets corresponding tothe respective references, the entire contents of which, including therelated patents and applications listed above and references listed inthe LIST OF REFERENCES, are incorporated herein by reference.

2. Discussion of the Background

Breast cancer is a leading cause of death in women, causing an estimated46,000 deaths per year. Mammography is the most effective method for theearly detection of breast cancer, and it has been shown that periodicscreening of asymptomatic women does reduce mortality. Many breastcancers are detected and referred for surgical biopsy on the basis of aradiographically detected mass lesion or cluster of microcalcifications.Although general rules for the differentiation between benign andmalignant mammographically identified breast lesions exist, considerablemisclassification of lesions occurs with the current methods. Onaverage, less than 30% of masses referred for surgical breast biopsy areactually malignant.

Accordingly, due shortcomings in the above-noted methods, an improvedmethod, system and computer readable medium for the computer assistedinterpretation of medical images is desirable.

SUMMARY OF THE INVENTION

Accordingly, an object of this invention is to provide a method, systemand computer readable medium that employs an intelligent searchworkstation for the computer assisted interpretation of medical images.

Another object of the invention is to provide an automated method,system and computer readable medium that employs an intelligent searchworkstation for the computer assisted interpretation of medical imagesbased on computer-estimated likelihood of a pathological state, e.g.,malignancy.

Another object of the invention is to provide a method, system andcomputer readable medium that employs an intelligent search workstationfor the computer assisted interpretation of medical images and outputsto the radiologist/physician output from the computer analysis of themedical images.

These and other objects are achieved according to the invention byproviding an improved method, system and computer readable medium forcomputer assisted interpretation of a medical image, including obtainingimage data representative of a medical image; computing at least onefeature characteristic of the image data; comparing the computed featurecharacteristic to corresponding computed feature characteristics derivedfrom images in a known image data set; selecting image data from imagesof the known image data set having corresponding computed featurecharacteristics similar to the feature characteristics computed in thecomputing step; displaying at least one of the selected image data andthe obtained image data.

The present invention accordingly includes a computer readable mediumstoring program instructions by which the method of the invention can beperformed when the stored program instructions are appropriately loadedinto a computer, and a system for implementing the method of theinvention.

The method, system computer readable medium of the intelligent searchworkstation combines the benefit of computer-aided diagnosis with priorknowledge obtained via confirmed cases. It is expected that the displayof known lesions with similar features will aid the radiologist inhis/her work-up of a suspect lesion, especially when the radiologist'sassessment of the lesion differs from the computer output, for thecomputerized assessment of tumor extent in magnetic resonance images.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the invention and many of the attendantadvantages thereof will be readily obtained as the same becomes betterunderstood by reference to the following detailed description whenconsidered in connection with the accompanying drawings wherein:

FIG. 1 is a flow chart illustrating the overall scheme for a method thatemploys an intelligent search workstation for the computer assistedinterpretation of medical images;

FIG. 2 is a graph illustrating the distribution of cases of mammographicmass lesions in the known database;

FIG. 3 is a graph illustrating the distribution of tumor pathology inthe database used in the demonstration and evaluation of the presentedmethod, system and computer readable medium;

FIG. 4 is a flow chart illustrating the method for computer calculationof the features and estimate of the likelihood of malignancy;

FIG. 5 is a graph illustrating the performance in terms of an ROC curveof the computer analysis on the database used as the reference databasein the intelligent search workstation, wherein the performance is givenfor the task of distinguishing malignant from benign lesions;

FIG. 6 is an image illustrating an example of the workstation interfacefor an unknown case using a color display where red indicates amalignant case and green indicates a benign case for a malignant case;

FIG. 7 is an image illustrating an example of the workstation interfacefor an unknown case using a color display where red indicates amalignant case and green indicates a benign case for a benign case;

FIG. 8 is a diagram illustrating the placement of the unknown case onthe probability distribution axes of the malignant and benign cases,wherein instead of just displaying typical malignant and benign casesthat are similar, the computer display shows relative similarity of themalignant and benign known cases by use of a color-coding of the similarlesions, whereby the probability distributions of the malignant andbenign cases in the known database are shown by images along with the“location” of the unknown case relative to the two distributions; and

FIG. 9 is a schematic illustration of a general purpose computer whichcan be programmed according to the teachings of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The inventors are investigating the potential usefulness ofcomputer-aided diagnosis as an aid to radiologists in thecharacterization and classification of mass lesions in mammography.Observer studies show that such a system can aid in increasing thediagnostic accuracy of radiologists both in terms of sensitivity andspecificity. The present mass classification method, system and computerreadable medium includes three components: (1) automated segmentation ofmass regions, (2) automated feature-extraction, and (3) automatedclassification [1–3]. The method is initially trained with 95 mammogramscontaining masses from 65 patients. Features related to the margin,shape, and density of each mass are extracted automatically from theimage data and merged into an estimate of the likelihood of malignancyusing artificial neural networks (ANNs). These features include aspiculation measure, a radial gradient index, and two density measures.The round-robin performance of the computer in distinguishing betweenbenign and malignant masses is evaluated by receiver operatingcharacteristic (ROC) analysis. The computer classification schemeyielded an A_(z) value of 0.94, similar to that of an experiencedmammographer (A_(z)=0.91) and statistically significantly higher thanthe average performance of five radiologists with less mammographicexperience (A_(z)=0.80). With the database used, the computer schemeachieved, at 100% sensitivity, a positive predictive value of 83%, whichis 12% higher than that of the experienced mammographer and 21% higherthan that of the average performance of the less experiencedmammographers at a p-value of less than 0.001.

The computerized mass classification method is next independentlyevaluated on a 110-case clinical database consisting of 50 malignant and60 benign cases. The effects of variations in both case mix and in filmdigitization technique on the performance of the method are assessed.Categorization of lesions as malignant or benign using the computerachieved an A_(z) value (area under the ROC curve) of 0.90 on the priortraining database (Fuji scanner digitization) in a round-robinevaluation, and A_(z) values of 0.82 and 0.81 on the independentdatabase for Konica and Lumisys digitization formats, respectively.However, the statistical comparison of these performances fails to showa statistical significant difference between the performance on thetraining database and that on the independent validation database(p-values>0.10). Thus, the computer-based method for the classificationof lesions on mammograms is shown to be robust to variations in case mixand film digitization technique.

The inventors have now developed an intelligent search display intowhich has been incorporated the computerized mass classification method.Upon viewing an unknown mammographic case, the display shows both thecomputer classification output as well as images of lesions with knowndiagnoses (e.g., malignant vs. benign) and similar computer-extractedfeatures. The similarity index used in the search can be chosen by theradiologist to be based on a single feature, multiple features, or onthe computer estimate of the likelihood of malignancy.

The output of a computer-aided diagnostic scheme can take a variety offorms such as the estimated likelihood that a lesion is malignant eitherin terms of probabilities or along a standardized rating scale. Thisinformation is then available for use by the radiologist as he or shesees fit when making decisions regarding patient management. Analternative approach is for the computer to display a variety of lesionsthat have characteristics similar to the one at band and for which thediagnosis is known, thereby providing a visual aid for the radiologistin decision making. Swett et al [4, 5] presented such a method in theMammo/Icon system which used human input characteristics of the lesionin question. Sklansky et al [6] uses a visual neural network anddisplays the feature plot of the known database with the unknown caseindicated. The intelligent workstation according to the presentinvention is unique in that it recalls lesions in the known databasebased either on a single feature, multiple features, orcomputer-estimate of the likelihood of malignancy. In addition, insteadof just displaying typical malignant and benign cases that are similar,the computer display shows relative similarity of the malignant andbenign known cases by use of a color-coding of the similar lesions.Basically, the probability distributions of the malignant and benigncases in the known database are shown by images along with the“location” of the unknown case relative to the two distributions.

The intelligent search workstation combines the benefit ofcomputer-aided diagnosis with prior knowledge obtained via confirmedcases. It is expected that the display of known lesions with similarfeatures will aid the radiologist in his/her workup of a suspect lesion,especially when the radiologist's assessment of the lesion differs fromthe computer output.

Referring now to the drawings, wherein like reference numerals designateidentical or corresponding parts throughout the several views, and moreparticularly to FIG. 1 thereof, there is illustrated a top level blockdiagram of the method that employs an intelligent search workstation forthe computer assisted interpretation of medical images according to thepresent invention.

The overall scheme includes an initial acquisition of a set of knownmedical images that comprise a database, and presentation of the imagesin digital format. The lesion location in terms of estimated center isinput from either a human or computer. The method, system and computerreadable medium that employs an intelligent search workstation for thecomputer assisted interpretation of medical images consists of thefollowing stages: access to a database of known medical images withknown/confirmed diagnoses of pathological state (step 112),computer-extraction of features of lesions within the known database(step 104), input method for an unknown case (step 102),computer-extraction of features of lesion of the unknown case (step104), calculation of likelihood of pathological state (e.g., likelihoodof malignancy) for the known and unknown cases (step 108), calculationof similarity indices for the unknown cases relative to each of theknown cases (step 110), search among the known database based on thecalculated similarity indices (step 114) and presentation of the“similar” cases and/or the computer-estimated features and/or likelihoodof pathological state (step 116). A specific example of the system isgiven for mass lesions in mammographic images in which the computerextracts features (step 104) and estimates the likelihood of malignancyfor the known and the unknown cases (step 108), computes the similarityindices for each pair (step 110), and output cases that are similar interms of individual features, combination of features, and/orcomputer-estimated likelihood of malignancy (step 116). The radiologisthas the option of choosing various features, such as single feature,multiple features, likelihood of malignancy, etc., for the analysis(step 106).

Database

The images used are obtained by mammography followed by filmdigitization. The database used consists of 95 clinical mammograms(Kodak MinR screen/OM-1 film, Eastman Kodak, Rochester, N.Y.), eachcontaining a mass. Of the ninety five mammographic masses, 38 are benignand 57 are malignant. All but one case is verified by biopsy; theremaining one is deemed benign by long-term follow-up as illustrated inthe graph of FIG. 2. The computerized mass classification method isindependently evaluated on a 110-case clinical database consisting of 50malignant and 60 benign cases as illustrated in the graph of FIG. 3.

Automated Classification of Mass Lesions

As shown in the flow chart of FIG. 4, the mass classification methodincludes three components: (1) automated segmentation of mass regionsincluding inputting of digital images (step 402), manual or automaticindication of lesion center (step 404), and automated lesion extraction(step 406), (2) automated feature-extraction (step 408) forpredetermined features, such as spiculation, shape, margin sharpness,texture, etc. (step 410), and (3) automated classification (step 412).[1–3]. After automated determination of the likelihood of malignancy ispreformed (step 414), the likelihood of malignancy and/or lesionfeatures output (step 416) are input to the intelligent workstation(step 418).

The segmentation (step 402) of a mass from the background parenchyma isaccomplished using a multiple-transition-point, gray levelregion-growing technique. Segmentation begins within a 512×512 pixelregion of interest manually or automatically centered about theabnormality in question (step 404). In clinical practice, the locationof the mass can be identified either by a radiologist or by acomputer-detection scheme and then fed into the classification schemefor an output on the likelihood of malignancy. In order to correct forthe nonuniformity of the background distribution and to enhance imagecontrast for better segmentation of masses, background trend correctionand histogram equalization techniques are applied to the 512×512 regionof interest.

The margin, shape and density of a mass are three major characteristicsused by radiologists in classifying masses. Different characteristics ofthese features are associated with different levels of probabilities ofmalignancy. In order to determine the likelihood of malignancyassociated with different margin and density characteristics, theinventors developed algorithms that extract (step 408), for example, twofeatures that characterize the margin of a mass (e.g., spiculation andmargin sharpness) and three features that characterize the density of amass (e.g., average gray level, contrast and texture).

Margin characteristics typically are very important discriminants indifferentiating between benign and malignant masses. In order todetermine the likelihood of malignancy of a mass based on its margin,two major margin characteristics—a spiculation measure and amargin-sharpness measure—are developed. Margin spiculation typically isthe most important indicator for malignancy with spiculated lesionshaving a greater than 90% probability of malignancy. Margin sharpnesstypically is also very important in determination of the benign vs.malignant nature of a mass—with an ill-defined margin indicatingpossible malignancy and a well-defined margin indicating likelybenignity. Only about 2% of well-defined masses are malignant.

The spiculation measure is obtained from an analysis of radial edgegradients. The spiculation measure evaluates the average angle (degrees)by which the direction of the maximum gradient at each point along themargin of a mass deviates from the radial direction, the directionpointing from the geometric center of the mass to the point on themargin. The actual measure is the full width at half-maximum (FWHM) ofthe normalized edge-gradient distribution calculated for a neighborhoodof the grown region of the mass with respect to the radial direction.This measure is able to quantify the degree of spiculation of a massprimarily because the direction of maximum gradient along the margin ofa spiculated mass varies greatly from its radial direction, whereas thedirection of the maximum gradient along the margin of a smooth mass issimilar to its radial direction.

The spiculation measure achieved a similar level of performance(A_(z)=0.88) to that of the experienced mammographer's spiculationratings (A_(z)=0.85) in terms of the ability to distinguish betweenbenign and malignant masses based solely on spiculation. [1]

The sharpness of the margin of a mass, for example, can be described aswell-defined, partially ill-defined or ill-defined. The average marginsharpness can be quantified by calculating the magnitude of the averagegradient along the margin of the mass. A well-defined margin has a largevalue for the average margin sharpness measure, whereas an ill-definedmargin has a small value.

Although the radiographic density of a mass may not be by itself aspowerful a predictor in distinguishing between benign and malignantmasses as its margin features, taken with these features, densityassessment typically can be extremely useful. The evaluation of thedensity of a mass is of particular importance in diagnosingcircumscribed, lobulated, indistinct, or obscured masses that are notspiculated.

In order to assess the density of a mass radiographically, introducedare three density-related measures (e.g., average gray level, contrast,and texture measure) that characterize different aspects of the densityof a mass. These measures are similar to those used intuitively byradiologists. Average gray level is obtained by averaging the gray levelvalues of each point within the grown region of a mass. Contrast is thedifference between the average gray level of the grown mass and theaverage gray level of the surrounding fatty areas (e.g., areas with graylevel values in the lower 20% of the histogram for the total surroundingarea). Texture is defined here as the standard deviation of the averagegradient within a mass and it is used to quantify patterns arising fromveins, trabeculae, and other structures that may be visible through alow-density mass, but not through a high-density mass. A mass of lowradiographic density should have low values of average gray level andcontrast, and a high value of the texture measure, whereas a mass ofhigh radiographic density should have high values of average gray leveland contrast, and a low value of the texture measure.

Classifier

Three automated classifiers (step 412) are investigated for the task ofmerging the computer-extracted features (step 408) into an estimate ofthe likelihood of malignancy (step 414): (1) a rule-based method; (2) anartificial neural network; and (3) a hybrid system (i.e., combination ofa one-step rule-based method and an artificial neural network). Indetermining the likelihood of malignancy (step 414) for the cases thathad both the medio-lateral-oblique and cranio-caudal views, themeasurements obtained from both views are considered and the one withthe higher likelihood of malignancy estimated by the computer (step 414)is used in the evaluation. For example, in these cases, a mass would beclassified as malignant if either one of the two views showed suspicioussigns, e.g., either one of the FWHM measures from its two viewssatisfied the cutoff on the FWHM measure.

A rule-based method adopts knowledge from experts into a set of simplerules. Certain criteria for differentiating between benign and malignantmasses have been established by expert mammographers. The rules employedin the present approach for spiculation, margin-sharpness and densitymeasures are based on these criteria. A two-step rule-based method isstudied for this database. Because of its clinical diagnosticsignificance, the spiculation measure is applied first in the rule-basedmethod. After the spiculation measure (e.g., FWHM) is applied toidentify spiculated masses (e.g., including some irregular masses) andcategorized them as malignant first, a second feature is applied tofurther characterize the masses in the non-spiculated category aspreviously discussed. In order to investigate the potential discriminantability of the spiculation measure along with all the possible secondaryfeatures, applied separately is each of the remaining four features—themargin-sharpness measure and the three density measures—after thespiculation measure. The threshold of the spiculation measure (e.g.,FWHM of 160 degrees) is determined based on the entire database. Thethresholds of the other four features are determined based on theremaining database only.

The ANN approach is quite different from the rule-based method. Insteadof using prespecified empirical algorithms based on prior knowledge,ANNs are able to learn from examples and therefore can acquire their ownknowledge through learning. Also, neural networks are capable ofprocessing large amounts of information simultaneously. Neural networkstypically do not, however, provide the user with explanations abouttheir decisions and may not be able to bring pre-existing knowledge intothe network. Here employed is a conventional three-layer, feed-forwardneural network with a back-propagation algorithm, which has been used inmedical imaging and medical decision making. The structure of the neuralnetwork, for example, includes four input units (e.g., each of whichcorresponded to a computer-extracted feature), two hidden units, and oneoutput unit.

To determine the ability of the neural network to generalize from thetraining cases and make diagnoses for cases that had not been includedin the database, employed is a round-robin method—also known as theleave-one-out method. In this method, all but one case is used to trainthe neural network. The single case that is left out is used to test theneural network. For the cases having both medio-lateral-oblique andcranio-caudal views, both images of the pair are left out in theround-robin training. The higher value of the two from the round-robintest is reported as the estimated likelihood of malignancy. Thisprocedure is repeated for all the cases.

Each classifier has its advantages and limitations. With rule-basedmethods, one can adopt pre-existing knowledge as rules. However, thereare limitations in the availability of knowledge and knowledgetranslation. Even the experts find it difficult to articulate particulartypes of “intuitive” knowledge, and the process of translatingparticular knowledge into rules is limited by this expressive power.ANNs are capable of learning from examples and therefore can acquiretheir own knowledge. It may be most advantageous to use ANNs whenintuitive knowledge cannot be explicitly expressed or is difficult totranslate. However, the ANN requires a sufficiently large database tolearn effectively. Also, with an ANN there may be uncertainty as towhether the final learning goal is achieved in some situations. To takeadvantage of both rule-based systems and ANNs in the task of classifyingmasses, integrated is a rule-based method and an ANN into a hybridsystem. In the hybrid system, initially is applied a rule on thespiculation measure since both spiculated and irregular masses arehighly suspicious for malignancy, and then is applied an ANN to theremaining masses. Basically, this frees the ANN from having to “learn”the significance of spiculation to the detriment of learning thesignificance of the other features.

The threshold of the spiculation measure for the hybrid system is thesame as the one used in the rule-based method. The ANN applied in thehybrid system is a three-layer, feed-forward neural network with aback-propagation algorithm that has a structure of three input units(e.g., corresponding to the three remaining features used in the ANNmethod), two hidden units, and one output unit. The same round-robinmethod is applied to test the generalization ability of this neuralnetwork to differentiate between benign and malignant masses in thenon-spiculated category.

Evaluation of Automated Classification of Mass Lesions

The method is initially trained with 95 mammograms containing massesfrom 65 patients. Features related to the margin, shape, and density ofeach mass are extracted automatically from the image data and mergedinto an estimate of the likelihood of malignancy using artificial neuralnetworks. These features include a spiculation measure, a radialgradient index, and two density measures. The round-robin performance ofthe computer in distinguishing between benign and malignant masses isevaluated by receiver operating characteristic (ROC) analysis. As shownin FIG. 5, the computer classification scheme based on the hybrid systemyielded an A_(z) value of 0.94, similar to that of an experiencedmammographer (A_(z)=0.91) and the computer classification scheme basedon the ANN system (A_(z)=0.90) and statistically significantly higherthan the average performance of five radiologists with less mammographicexperience (A_(z)=0.80). With the database used, the computer schemeachieved, at 100% sensitivity, a positive predictive value of 83%, whichis 12% higher than that of the experienced mammographer and 21% higherthan that of the average performance of the less experiencedmammographers at a p-value of less than 0.001.

The computerized mass classification method is next independentlyevaluated on a 110-case clinical database consisting of 50 malignant and60 benign cases. The effects of variations in both case mix and in filmdigitization technique on the performance of the method are assessed.Categorization of lesions as malignant or benign using the computerachieved an A_(z) value (area under the ROC curve) of 0.90 on the priortraining database (i.e., Fuji scanner digitization) in a round-robinevaluation, and A_(z) values of 0.82 and 0.81 on the independentdatabase for Konica and Lumisys digitization formats, respectively.However, the statistical comparison of these performances failed to showa statistical significant difference between the performance on thetraining database and that on the independent validation database(p-values>0.10). Thus, the present computer-based method for theclassification of lesions on mammograms is shown to be robust tovariations in case mix and film digitization technique.

Integration of Automated Mass Lesion Classification into the IntelligentWorkstation

The output of the automated classification method (FIG. 4, step 416) isused to intelligently search for similar lesions in the known database(FIG. 4, step 418 and FIG. 1 step 114). FIG. 6 shows an example of theworkstation graphical user interface (GUI) for an unknown case 602using, for example, a color display where a red border 604 indicates amalignant case and green border 606 indicates a benign case for a casedeemed malignant by the computer scheme.

FIG. 7 shows an example of the workstation GUI for an unknown case 702using, for example, a color display where a red border 704 indicates amalignant case and a green border 706 indicates a benign case for a casedeemed benign by the computer scheme. Similar displays can be given forblack and white displays by, for example, outlining the known malignantlesions in black and outlining the known benign lesions in white.

Referring to FIGS. 6 and 7, the user specifies how the search 608/708 isto occur—either by a single feature, such as likelihood of malignancy608 a/708 a, spiculation 608 b/708 b, radial gradient 608 c/708 c,margin sharpness 608 d/708 d, average gray level 608 e/708 e, texture608 f/708 f, etc., by a combination of features (in which, e.g., theEuclidean distance is calculated for the combination of features betweenthe unknown case and each of the known cases), or by all of the features608 g/708 g. The display in the FIGS. 6 and 7, for example, is a searchby spiculation and under each known case the Euclidean distance measureas well as the spiculation measure 610/710 is given. The user has theoption to see the computer outputs, the similar known cases, or both.

The workstation interface shown in FIGS. 6 and 7 further includes acalculate button 612/712, file/edit/select drop down menus 614/714, anopen button 616/716, an exit button 618/718, sharpness/brightness radiobuttons 620/720, an information box 622/722 including yes/no buttons 622a/722 a for showing similarities and CAD results, a display more imagesbutton 624/724 and a help drop down menu 626/726.

FIG. 8 further explains the search method by schematically showing thelocation of an unknown case 802 (e.g., which may be shown in blue)relative to the distribution of malignant 804 (e.g., which may be shownin red) and benign 806 (e.g., which may be shown in green) cases withrespect to a given feature (and/or combination of features). The outputof the cases can be in terms of those closest (in terms of absolutedistance) as shown in FIGS. 6 and 7 or in terms of actual distance bynot employing an absolute-type similarity index. For example, if onlyspiculation is chosen for searching, the unknown cases can be displayedwith known cases on either side of it indicating which of the knowncases are more spiculated and which ones are less spiculated.

Observer Study Results

At the Radiological Society of North America (RSNA) 2000, the inventorsran a demonstration of the intelligent search workstation of the presentinvention to show how it helps radiologists. The demo included 20mammographic cases. For each case, the radiologists first saw themammograms including the standard views of MLO, where MLO ismedial-lateral oblique view (mammographic view of the breast at a sideways oblique angle) and CC, where CC is a cranial-caudal view(mammographic view of the breast from the head downwards), any specialviews of MLO and CC, as well as a 4 on 1 view showing the location ofthe lesion in the breast, where the 4 on 1 view is a collage of the leftand right CC views on top of the left and right MLO views. Eachradiologist was then asked to give his/her likelihood of malignancy on ascale from zero to 100, and also give his/her recommendation for eitherbiopsy or follow-up. Then the computer output of the likelihood ofmalignancy was shown along with the resulting images from theintelligent search from the reference library. Each radiologist then wasagain asked for his/her likelihood of malignancy and recommendation.After 20 cases, the sensitivity and specificity was calculated based ontheir biopsy/follow-up recommendation, where:

Sensitivity is the percentage of malignant (i.e., cancerous) cases sentto biopsy. So a sensitivity of 0.92 is 92%. The higher the sensitivitythe better.

Specificity is determined by subtracting the false-positive rate FPR,i.e., the number of benign cases sent to biopsy, from unity, i.e.,(1-FPR). The higher the specificity, the better.

TABLE 1 RSNA 2000 Exhibit Observer Study Results on Mass LesionClassification (Biopsy vs. Follow-Up) Spec- Spec- Sensitivity ificityificity Number of (w/o Sensitivity (w/o (w/ Reader Type Observers CAD)(w/ CAD) CAD) CAD) Mammographers 29 0.81 0.83 0.62 0.67 Radiologist Non-13 0.79 0.85 0.55 0.74 Mammographers

In Table 1 above, “with CAD” includes the computer output of thelikelihood of malignancy and the results of intelligent search on aknown atlas of images, using the intelligent search workstation of thepresent invention. From Table 1, one sees that both the sensitivity andthe specificity increased for mammographers and for radiologists (who donot read mammograms regularly) using the intelligent search workstationof the present invention.

FIG. 9 is a schematic illustration of a general purpose computer 900which can be programmed according to the teachings of the presentinvention. In FIG. 9, the computer 900 implements the processes of thepresent invention, wherein the computer includes, for example, a displaydevice 902 (e.g., a touch screen monitor with a touch-screen interface,etc.), a keyboard 904, a pointing device 906, a mouse pad or digitizingpad 908, a hard disk 910, or other fixed, high density media drives,connected using an appropriate device bus (e.g., a SCSI bus, an EnhancedIDE bus, an Ultra DMA bus, a PCI bus, etc.), a floppy drive 912, a tapeor CD ROM drive 914 with tape or CD media 916, or other removable mediadevices, such as magneto-optical media, etc., and a mother board 918.The mother board 918 includes, for example, a processor 920, a RAM 922,and a ROM 924 (e.g., DRAM, ROM, EPROM, EEPROM, SRAM, SDRAM, and FlashRAM, etc.), I/O ports 926 which may be used to couple to an imageacquisition device and optional special purpose logic devices (e.g.,ASICs, etc.) or configurable logic devices (e.g., GAL andre-programmable FPGA) 928 for performing specialized hardware/softwarefunctions, such as sound processing, image processing, signalprocessing, neural network processing, automated classification, etc., amicrophone 930, and a speaker or speakers 932.

As stated above, the system includes at least one computer readablemedium. Examples of computer readable media are compact discs, harddisks, floppy disks, tape, magneto-optical disks, PROMs (EPROM, EEPROM,Flash EPROM), DRAM, SRAM, SDRAM, etc. Stored on any one or on acombination of computer readable media, the present invention includessoftware for controlling both the hardware of the computer 900 and forenabling the computer 900 to interact with a human user. Such softwaremay include, but is not limited to, device drivers, operating systemsand user applications, such as development tools. Such computer readablemedia further includes the computer program product of the presentinvention for performing any of the processes according to the presentinvention, described above. The computer code devices of the presentinvention can be any interpreted or executable code mechanism, includingbut not limited to scripts, interpreters, dynamic link libraries, Javaclasses, and complete executable programs, etc.

The programming of general purpose computer 900 may include a softwaremodule for digitizing and storing images obtained from an imageacquisition device. Alternatively, the present invention can also beimplemented to process digital data derived from images obtained byother means, such as a picture archive communication system (PACS). Inother words, often the digital images being processed will be inexistence in digital form and need not be converted to digital form inpracticing the invention.

Accordingly, the mechanisms and processes set forth in the presentdescription may be implemented using a conventional general purposemicroprocessor or computer programmed according to the teachings in thepresent specification, as will be appreciated by those skilled in therelevant art(s). Appropriate software coding can readily be prepared byskilled programmers based on the teachings of the present disclosure, aswill also be apparent to those skilled in the relevant art(s). However,as will be readily apparent to those skilled in the art, the presentinvention also may be implemented by the preparation ofapplication-specific integrated circuits or by interconnecting anappropriate network of conventional component circuits.

The present invention thus also includes a computer-based product whichmay be hosted on a storage medium and include instructions which can beused to program a general purpose microprocessor or computer to performprocesses in accordance with the present invention. This storage mediumcan include, but is not limited to, any type of disk including floppydisks, optical disks, CD-ROMs, magneto-optical disks, ROMs, RAMs,EPROMs, EEPROMs, flash memory, magnetic or optical cards, or any type ofmedia suitable for storing electronic instructions.

Recapitulating, the method, system and computer readable medium employsan intelligent search workstation for the computer assistedinterpretation of medical images. The method, system and computerreadable medium of the intelligent search interface takes as input anunknown medical image and searches through a set of cases with knowndiagnoses by way of a similarity index. The interface then displaysimages having similar features in a specified order. The search isdirected by computer-extracted features (although it can easily beadapted for human-extracted features) and an estimate of the likelihoodof the diagnosis of the unknown lesion is obtained with use of anartificial neural network (ANN, although others classifiers can beused). The similarity index is calculated using an Euclidean distancemeasure (although other measures of similarity can be employed). Thedisplayed images can have their diagnostic truth displayed by means ofenclosure by a green or red outline (for benign or malignant,respectively). In a sense, the display is showing the distribution ofmalignant and benign cases in the known database (or databank) that aresimilar to the unknown case. The significance of this method, system andcomputer readable medium is in the diagnosis of lesions and medicalconditions, especially for confusing cases. The method, system andcomputer readable medium can also be used together with computerizedoutput of the features and likelihood of malignancy (computer-aideddiagnosis).

Although the present invention is described in terms of practicing themethod on mammographic image data sets, the intelligent workstation canbe implemented for other medical images, such as chest radiography,ultrasound, magnetic resonance imaging, etc., in which a computerizedanalysis of image or lesion features is performed with respect to somedisease state, as will be appreciated by those skilled in the relevantart(s).

Numerous modifications and variations of the present invention arepossible in light of the above teachings. It is therefore to beunderstood that within the scope of the appended claims, the inventionmay be practiced otherwise than as specifically described herein.

LIST OF REFERENCES

-   [1]. Huo Z, Giger M L, Vyborny C J, Bick U, Lu P, Wolverton D E,    Schmidt R A: Analysis of spiculation in the computerized    classification of mammographic masses” Medical Physics 22:1569–1579,    1995.-   [2]. Huo Z, Giger M L, Vyborny C J, Wolverton D E, Schmidt R A, Doi    K: Automated computerized classification of malignant and benign    mass lesions on digitized mammograms. Academic Radiology 5: 155–168,    1998.-   [3]. Huo Z, Giger M L, Metz C E: Effect of dominant features on    neural network performance in the classification of mammographic    lesions. PMB 44: 2579–2595, 1999.-   [4]. Swett H A, Fisher P R, Cohn A I, et al: Expert system    controlled image display. Radiology 172:487, 1989-   [5]. Swett H A, Miller P A. ICON: a computer-based approach to    differential diagnosis in radiology. Radiology 163:555, 1987-   [6]. Sklansky J, Taso E Y, Ornes C, Disher A C. A visualized    mammographic database in computer-aided diagnosis. Computer-Aided    Diagnosis in Medical Imaging. Eds Doi K, MacMahon, Giger M L,    Hoffmann K R. Pp. 215–220. Elsevier Science, 1999.

1. In a user interface of a computer-aided detection (CAD) system, amethod for facilitating user diagnosis of a detected lesion in an x-raymammogram, comprising: displaying a first image of the detected lesionto the user; selecting a plurality of reference images from a referencelesion database based on similarity to said detected lesion with respectto at least one computed feature characteristic; and displaying saidplurality of reference images near the first image, each reference imagecomprising a lesion with a known diagnosis and being visibly markedaccording to said known diagnosis.
 2. The method of claim 1, comprising:displaying said reference images sized smaller than said first image andsimultaneously with display of said first image.
 3. The method of claim2, wherein said at least one computed feature characteristic comprises:a likelihood-of-malignancy metric computed by the CAD system.
 4. Themethod of claim 2, wherein said at least one computed featurecharacteristic is selected from the group consisting of: a marginsharpness metric; a spiculation metric; a mass density metric; and alikelihood-of-malignancy metric computed by the CAD system.
 5. Themethod of claim 2, comprising: spatially ordering said reference imageson said user interface according to said computed featurecharacteristic.
 6. The method of claim 1, wherein the detected lesion isa CAD-detected lesion selected by the user from a CAD-annotated x-raymammogram display.
 7. In a user interface of a computer-aided detection(CAD) system, a method for facilitating user diagnosis of a detectedlesion in an x-ray mammogram, comprising: receiving a selection of thedetected lesion on a first display; displaying a first image of thedetected lesion on a second display; and displaying a plurality ofreference images on the second display near the first image, eachreference image based on a similarity to said detected lesion withrespect to at least one computed feature characteristic and comprising alesion with a known diagnosis and being visibly marked according to saidknown diagnosis, including displaying said reference images sizedsmaller than said first image and simultaneously with display of saidfirst image on said second display.
 8. The method of claim 7,comprising: selecting said plurality of reference images from areference lesion database based on similarity to said detected lesionwith respect to at least one computed feature characteristic.
 9. Themethod of claim 8, wherein said at least one computed featurecharacteristic comprises: a likelihood-of-malignancy metric computed bythe CAD system.
 10. The method of claim 8, wherein said at least onecomputed feature characteristic is selected from the group consisting ofa margin sharpness metric, a spiculation metric, a mass density metric,and a likelihood-of-malignancy metric computed by the CAD system. 11.The method of claim 8, comprising: spatially ordering said referenceimages on said user interface according to said computed featurecharacteristic.
 12. In a user interface of a computer-aided detection(CAD) system, a method for facilitating user diagnosis of a detectedlesion in an x-ray mammogram, comprising: displaying a first image ofthe detected lesion to the user; displaying a plurality of referenceimages near the first image, each reference image based on a similarityto said detected lesion with respect to at least one computed featurecharacteristic and comprising a lesion with a known diagnosis and beingvisibly marked according to said known diagnosis, including spatiallyordering said reference images on said user interface according to adegree of similarity with said detected lesion with respect to at leastone computed feature characteristic.
 13. The method of claim 12, whereinsaid at least one computed feature characteristic comprises: alikelihood-of-malignancy metric computed by the CAD system.
 14. Themethod of claim 12, wherein said at least one computed featurecharacteristic is selected from the group consisting of a marginsharpness metric, a spiculation metric, a mass density metric, and alikelihood-of-malignancy metric computed by the CAD system.
 15. Themethod of claim 12, comprising displaying said reference images sizedsmaller than said first image and simultaneously with display of saidfirst image.